Driving assistance device, driving assistance method, and storage medium

ABSTRACT

Provided is a driving assistance device including a storage medium storing computer-readable commands and a processor connected to the storage medium, the processor executing the computer-readable commands to: recognize an external object near a mobile object; estimate toward which gaze candidate an occupant of the mobile object is directing his or her gaze from among a plurality of gaze candidates including gaze candidates existing on a structure of the mobile object and the external object; determine that the occupant visually recognizes a gaze candidate from among the plurality of gaze candidates when the occupant is estimated to direct his or her gaze toward the gaze candidate for a first predetermined period or more; and execute driving assistance for the mobile object based on a determination result.

CROSS-REFERENCE TO RELATED APPLICATION

The application is based on Japanese Patent Application No. 2022-087539filed on May 30, 2022, the content of which incorporated herein byreference.

BACKGROUND Field of the Invention

The present invention relates to a driving assistance device, a drivingassistance method, and a storage medium.

Description of Related Art

Hitherto, a technology for detecting the gaze of an occupant of a mobileobject and assisting the occupant in driving the mobile object based onthe detected gaze has been known. For example, Patent Publication No.2021-33570 discloses a technology for restricting reception to a controlpanel of a mobile object when the mobile object is traveling and thedetected gaze is not directed toward a predetermined area.

However, the related art cannot flexibly execute driving assistanceaccording to the direction or period of gaze of the occupant of themobile object in some cases.

SUMMARY

The present invention has been made in view of the above circumstances,and has an object to provide a driving assistance device, a drivingassistance method, and a storage medium that are capable of flexiblyexecuting driving assistance according to the direction or period ofgaze of the occupant of a mobile object.

The driving assistance device, the driving assistance method, and thestorage medium according to the present invention adopt the followingconfiguration.

(1): According to one aspect of the present invention, there is provideda driving assistance device including a storage medium storingcomputer-readable commands and a processor connected to the storagemedium, the processor executing the computer-readable commands to:recognize an external object near a mobile object; estimate toward whichgaze candidate an occupant of the mobile object is directing his or hergaze from among a plurality of gaze candidates including gaze candidatesexisting on a structure of the mobile object and the external object;determine that the occupant visually recognizes a gaze candidate fromamong the plurality of gaze candidates when the occupant is estimated todirect his or her gaze toward the gaze candidate for a firstpredetermined period or more; and execute driving assistance for themobile object based on a determination result, in which when theprocessor has estimated that the occupant has turned his or her gazeaway from the gaze candidate after determining that the occupantvisually recognizes the gaze candidate, the processor keeps thedetermination result, which indicates that the occupant visuallyrecognizes the gaze candidate determined to be gazed at, for a secondpredetermined period since a time point at which the occupant isestimated to have turned his or her gaze away from the gaze candidate.

(2): In the aspect (1), the processor estimates toward which gazecandidate the occupant of the mobile object is directing his or her gazefrom among the plurality of gaze candidates based on a degree of matchbetween the gaze and a gaze probability distribution representingprobabilities of gaze toward each gaze candidate.

(3): In the aspect (2), the processor calculates the degree of matchbased on the gaze represented by polar coordinates centered around ahead of the occupant of the mobile object, and a center angle and anglewidth of the gaze candidate represented by polar coordinates centeredaround the head, and estimates toward which gaze candidate the occupantof the mobile object is directing his or her gaze from among theplurality of gaze candidates.

(4): In the aspect (1), the processor changes a length of the secondpredetermined period according to types of the plurality of gazecandidates.

(5): In the aspect (1), the processor increases the second predeterminedperiod as a period in which the occupant is determined to visuallyrecognize the gaze candidate becomes longer.

(6): In the aspect (1), the processor increases the second predeterminedperiod as the number of times the occupant visually recognizes therecognized external object becomes larger.

(7): In the aspect (2), the processor changes a length of the secondpredetermined period according to a level of the calculated degree ofmatch.

(8): In the aspect (1), when it is determined that the occupant visuallyrecognize the gaze candidate and visual recognition of the gazecandidate enables recognition of an object included in the gazecandidate, the processor execute the driving assistance so as to reducea degree of alertness for the object.

(9): According to another aspect of the present invention, there isprovided a driving assistance method to be executed by a computer, thedriving assistance method including: recognizing an external object neara mobile object; estimating toward which gaze candidate an occupant ofthe mobile object is directing his or her gaze from among a plurality ofgaze candidates including gaze candidates existing on a structure of themobile object and the external object; determining that the occupantvisually recognizes a gaze candidate from among the plurality of gazecandidates when the occupant is estimated to direct his or her gazetoward the gaze candidate for a first predetermined period or more;executing driving assistance for the mobile object based on adetermination result; and keeping, when it is estimated that theoccupant has turned his or her gaze away from the gaze candidate afterdetermining that the occupant visually recognizes the gaze candidate,the determination result, which indicates that the occupant visuallyrecognizes the gaze candidate determined to be gazed at, for a secondpredetermined period since a time point at which the occupant isestimated to have turned his or her gaze away from the gaze candidate.

(10): According to another aspect of the present invention, there isprovided a non-transitory computer-readable storage medium storing aprogram for causing a computer to: recognize an external object near amobile object; estimate toward which gaze candidate an occupant of themobile object is directing his or her gaze from among a plurality ofgaze candidates including gaze candidates existing on a structure of themobile object and the external object; determine that the occupantvisually recognizes a gaze candidate from among the plurality of gazecandidates when the occupant is estimated to direct his or her gazetoward the gaze candidate for a first predetermined period or more;execute driving assistance for the mobile object based on adetermination result; and keep, when it is estimated that the occupanthas turned his or her gaze away from the gaze candidate afterdetermining that the occupant visually recognizes the gaze candidate,the determination result, which indicates that the occupant visuallyrecognizes the gaze candidate determined to be gazed at, for a secondpredetermined period since a time point at which the occupant isestimated to have turned his or her gaze away from the gaze candidate.

According to the aspects (1) to (10), it is possible to flexibly executedriving assistance according to the direction or period of gaze of theoccupant of a mobile object.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a configuration diagram illustrating a vehicle M including adriving assistance device according to an embodiment.

FIG. 2 is a diagram illustrating an exemplary position at which a drivermonitoring camera is placed in the vehicle including the drivingassistance device.

FIG. 3 is a diagram for describing a method of detecting the gaze of adriver by a gaze estimation unit.

FIG. 4 is a diagram for describing a method of calculating a centerangle and width of a gaze candidate by the gaze estimation unit.

FIG. 5 is a diagram for describing a method of determining that thedriver has visually recognized a gaze candidate by a gaze determinationunit.

FIG. 6 is a diagram for describing a method of calculating a secondpredetermined period in consideration of a lap ratio by the gazedetermination unit.

FIG. 7 is a diagram for describing driving assistance to be executed bya driving assistance unit.

FIG. 8 is a diagram for describing driving assistance to be executed bythe driving assistance unit.

FIG. 9 is a flow chart illustrating an exemplary flow of operations tobe executed by the driving assistance device.

DESCRIPTION OF EMBODIMENTS

Now, a driving assistance device, a driving assistance method, and astorage medium according to an embodiment of the present invention aredescribed with reference to the drawings.

[Overall Configuration]

FIG. 1 is a configuration diagram illustrating a vehicle M including adriving assistance device 100 according to an embodiment. The vehicle Mis, for example, a vehicle such as a two-wheeled vehicle, athree-wheeled vehicle, or a four-wheeled vehicle. The power source ofthe vehicle M is an internal combustion engine such as a diesel engineor a gasoline engine, an electric motor, or a combination thereof. Theelectric motor operates by using power generated by a generatorconnected to the internal combustion engine or power discharged by asecondary battery or a fuel cell. The vehicle M is an example of “mobileobject”.

The vehicle M includes, for example, a camera 10, a radar device 12, aLIDAR (Light Detection and Ranging) device 14, an object recognitiondevice 16, a communication device 20, an HMI (Human Machine Interface)30, a vehicle sensor 40, a navigation device 50, a driver monitoringcamera 60, a driving controller 70, a driving assistance device 100, adriving force output device 200, a braking device 210, and a steeringdevice 220. These devices and instruments are connected to one anothervia, for example, a wireless communication line, a serial communicationline, or a multiplex communication line such as a CAN (Controller AreaNetwork) communication line. The configuration illustrated in FIG. 1 isonly one example, and a part of the configuration may be omitted, oranother configuration may be added thereto.

The camera 10 is, for example, a digital camera that uses a solid imagepickup device such as a CCD (Charge Coupled Device) device or a CMOS(Complementary Metal Oxide Semiconductor) device. The camera 10 ismounted on any part of a vehicle (hereinafter referred to as “vehicleM”) having the vehicle system 1 mounted thereon. When the camera 10picks up a front image, the camera 10 is mounted on, for example, anupper part of a front windshield or a back surface of a rear-viewmirror. The camera 10 repeatedly photographs the surroundings of thevehicle M periodically, for example. The camera 10 may be a stereocamera.

The radar device 12 radiates a radio wave such as a millimeter wavetoward the surroundings of the vehicle M, and detects a radio wave(reflected wave) reflected by an object, to detect at least the position(distance and direction) of the object. The radar device 12 is mountedon any part of the vehicle M. The radar device 12 may detect theposition and speed of the object by an FM-CW (Frequency ModulatedContinuous Wave) method.

LIDAR 14 emits light (or electromagnetic wave with wavelength close tolight) around the vehicle M and measures the scattered light. The LIDAR14 detects a distance to the object based on the time from emission oflight to reception of light. The emitted light is, for example, a pulsedlaser light. The LIDAR 14 is mounted on any part of the vehicle M.

The object recognition device 16 executes sensor fusion processing forresults of detection by the camera 10, the radar device 12, and theLIDAR 14, to thereby recognize a position, a type, and a speed of anobject. The object recognition device 16 outputs the recognition resultto the driving assistance device 100. The object recognition device 16may output the results of detection by the camera 10, the radar device12, and the LIDAR 14, to the driving assistance device 100 as it is. Theobject recognition device 16 may be omitted from the vehicle system 1.

The HMI 30 presents various kinds of information to the occupants of thevehicle M and accepts input operations by the occupants. The HMI 30includes various display devices, speakers, buzzers, vibrationgenerators (vibrators), touch panels, switches, keys, etc.

The vehicle sensor 40 includes a vehicle speed sensor that detects thespeed of the vehicle M, an acceleration sensor that detects anacceleration, a yaw rate sensor that detects an angular speed around avertical axis, an azimuth sensor that detects the orientation of thevehicle M, and other sensors.

The navigation device 50 includes, for example, a GNSS (GlobalNavigation Satellite System) receiver, a guidance control unit, and astorage unit storing map information. The GNSS receiver identifies theposition of the vehicle M based on a signal received from a GNSSsatellite. The position of the vehicle M may be identified orsupplemented by an inertial navigation system (INS) using the output ofthe vehicle sensor 40. The guidance control unit, for example,determines the route from the position of the vehicle M identified bythe GNSS receiver (or any input position) to the destination entered bythe occupant by referring to the map information, and causes the HMI 30to output guidance information so that the vehicle M travels along theroute. The map information is, for example, information in which roadgeometry is represented by links indicating roads and nodes connected bythe links. The map information may include road curvature and POI (PointOf Interest) information, etc. The navigation device 50 may transmit thecurrent position and destination of the vehicle M to a navigation servervia a communication device and obtain a route from the navigationserver.

The driver monitoring camera 60 is, for example, a digital camera thatuses a solid image pickup device such as a CCD (Charge Coupled Device)device or a CMOS (Complementary Metal Oxide Semiconductor) device. Thedriver monitoring camera 60 is mounted on any part of the vehicle M at aposition and in an orientation that enable the head of an occupantseated in the driver's seat of the vehicle M (hereinafter referred to as“driver”) to be photographed from the front (in an orientation thatenables the face to be photographed). The driver monitoring camera 60outputs, to the driving assistance device 100, the image obtained byphotographing the cabin of the vehicle M including the driver from theposition at which the driver monitoring camera 60 is placed.

FIG. 2 is a diagram illustrating an exemplary position at which thedriver monitoring camera 60 is placed in the vehicle M including thedriving assistance device 100. In FIG. 2 , RVM represents a rearviewmirror, FWS represents a front windshield, LSM represents a left sideview mirror, LWS represents a left windshield, RSM represents a rightside view mirror, RWS represents a right windshield, DB represents adashboard, SW represents a steering wheel, and OB represents an object(pedestrian) outside the vehicle recognized by the object recognitiondevice 16. As illustrated in FIG. 2 , the driver monitoring camera 60 ismounted on a lower side of the display device (HMI 30) provided at thecenter of the dashboard of the vehicle M, for example, and photographsthe head of the driver from the front. The RVM rearview mirror, thefront windshield FWS, the left side view mirror LSM, the left windshieldLWS, the right side view mirror RSM, the right windshield RWS, the dashboard DB, and the object OB are examples of “gaze candidate”, and a gazeestimation unit 120 described later determines toward which one of thegaze candidates the driver is directing his or her gaze.

The driving controller 70 includes, for example, an acceleration pedal,a brake pedal, a shift lever, a steering wheel, and other controllers. Asensor for detecting the operation amount or whether or not an operationis performed is mounted to the driving controller 70, and the detectionresult is output to a part or all of the driving force output device200, the braking device 210, and the steering device 220.

The driving force output device 200 outputs, to drive wheels, a drivingforce (torque) for causing the subject vehicle to travel. The drivingforce output device 200 includes, for example, a combination of aninternal combustion engine, an electric motor, a transmission, and thelike, and a ECU (Electronic Control Unit) for controlling thesecomponents. The ECU controls the above-mentioned components according toinformation input from the driving assistance device 100 or informationinput from the driving controller 70.

The braking device 210 includes, for example, a brake caliper, acylinder for transmitting hydraulic pressure to the brake caliper, anelectric motor for causing hydraulic pressure in the cylinder, and anECU. The ECU controls the electric motor according to information inputfrom the driving assistance device 100 or information input from thedriving controller 70, and causes a brake torque that depends on abraking operation to be output to each wheel. The brake device 210 maybe equipped with, as a backup, a mechanism that transmits the hydraulicpressure generated by the operation of the brake pedal included in thedriving controller 70 to a cylinder via the master cylinder. The brakedevice 210 is not limited to have the above configurations, and may bean electronic control hydraulic brake device for controlling an actuatoraccording to information input from the driving assistance device 100and transmitting the hydraulic pressure of the master cylinder to thecylinder.

The steering device 220 includes, for example, a steering ECU and anelectric motor. The electric motor applies a force to a rack and pinionmechanism to change the orientation of a steered wheel, for example. Thesteering ECU drives the electric motor to change the orientation of thesteered wheel according to information input from the driving assistancedevice 100 or information input from the driving controller 70.

[Driving Assistance Device]

The driving assistance device 100 includes, for example, an objectrecognition unit 110, a gaze estimation unit 120, a gaze determinationunit 130, and a driving assistance unit 140. These components areimplemented by a hardware processor such as a CPU (Central ProcessingUnit) executing a program (software), for example. Further, a part orall of these components may be implemented by hardware (circuit unitincluding circuitry) such as an LSI (Large Scale Integration), an ASIC(Application Specific Integrated Circuit), an FPGA (Field-ProgrammableGate Array), or a GPU (Graphics Processing Unit), or may be implementedthrough cooperation between software and hardware. The program may bestored in a storage device (storage device including non-transitorystorage medium) such as an HDD or flash memory in advance, or may bestored in a removable storage medium such as a DVD or CD-ROM and thestorage medium (non-transitory storage medium) may be attached to adrive device to install the program into the HDD or flash memory of thedriving assistance device 100.

The object recognition unit 110 recognizes an object existing near thevehicle M based on the recognition results by the object recognitionunit 16. The object recognition unit 110 may recognize any objectexisting near the vehicle M or may recognize only a moving object (riskobject) existing near the vehicle M using, for example, the radar system12 and/or the LIDAR 14.

The gaze estimation unit 120 uses the images output by the camera 10 andthe driver monitoring camera 60 to estimate toward which gaze candidatethe driver of the vehicle M is directing his or her gaze from among aplurality of gaze candidates including gaze candidates existing on astructure of the vehicle M and the object recognized by the objectrecognition unit 110.

FIG. 3 is a diagram for describing a method of detecting the gaze LS ofthe driver by the gaze estimation unit 120. The gaze estimation unit 120first acquires an image including the driver of vehicle M photographedby the driver monitor camera 60, and then detects the gaze of the driverby any known algorithm (e.g., corneal reflection method). The gazeestimation unit 120 represents the detected gaze by polar coordinateswith, for example, the driver's head as a center O. Alternatively, thegaze estimation unit 120 may represent the gaze by polar coordinateswith, for example, the center O between the driver's eyebrows, or moregenerally, it suffices that the center O be set near the driver's eyes.In the following, the polar coordinates of the gaze LS detected by thegaze estimation unit 120 at a time t are represented by s_(t)(φ_(s_t),θ_(s_t)).

Further, when the gaze estimation unit 120 has acquired an imageincluding the object photographed by the camera 10, the gaze estimationunit 120 calculates the center angle and angle width of a plurality ofgaze candidates including the object, which are represented by polarcoordinates with the driver's head as the center O. As described above,the gaze candidate represents a candidate being gazed by the driver ofthe vehicle M, and includes, for example, the rearview mirror RVM, thefront windshield FWS, the left side view mirror LSM, the left windshieldLWS, the right side view mirror RSM, the right windshield RWS, the dashboard DB, and the object OB.

FIG. 4 is a diagram for describing a method of calculating the centerangle and width of the gaze candidate by the gaze estimation unit 120.As an example, FIG. 4 represents an example in which the gaze estimationunit 120 calculates the center angle and width of the object OB. In FIG.4 , RP represents a reference point (for example, point representingobject OB such as center of gravity). The gaze estimation unit 120calculates the reference point RP in polar coordinates. In thefollowing, the polar coordinates of the reference point RP of an n-thgaze candidate calculated by the gaze estimation unit 120 at a timepoint t are represented as a center angle (φ_(obj_n,t), θ_(obj_n,t))Further, the gaze estimation unit 120 calculates the width (in otherwords, extent of angle (φ, θ) of object OB in vertical and horizontaldirections) of the angle of the object OB with respect to the center O.In the following, the polar coordinates of the width of the angle of ann-th gaze candidate calculated by the gaze estimation unit 120 at thetime point t are represented as an angle width (Φ_(obj_n,t),Θ_(obj_n,t)). Φ represents the angle width in the φ direction, and Θrepresents the angle width in the θ direction. The gaze estimation unit120 calculates the center angle and angle width with respect to thecenter O also for the rearview mirror RVM, the front windshield FWS, theleft side view mirror LSM, the left windshield LWS, the right side viewmirror RSM, the right windshield RWS, and the dash board DB. However,the center angle and angle width for these gaze candidates having afixed positional relationship from the driver's seat may be calculatedand stored in advance.

After the gaze estimation unit 120 calculates the polar coordinatess_(t)(φ_(s_t), θ_(s_t)) of the gaze LS and the center angle(φ_(obj_n,t), θ_(obj_n,t)) and the angle width (Φ_(obj_n,t),Θ_(obj_n,t)) for each gaze candidate at the time point t, the gazeestimation unit 120 inputs these values to a probability modelrepresented by the following expression (1) to calculate a probabilityvalue of the driver directing his or her gaze toward each gazecandidate.

[Math.1] $\begin{matrix}{{P( {{{targ} = {n❘s_{0:t}}},{obj}_{{1:N},{0:t}}} )} = \frac{{p( {{{s_{t}❘{targ}} = n},{obj}_{n,t}} )}{P( {{{targ} = {n❘s_{{0:t} - 1}}},{obj}_{{1:N},{{0:t} - 1}}} )}}{p( {{s_{t}❘s_{{0:t} - 1}},{obj}_{{1:N},{0:t}}} )}} & (1)\end{matrix}$

The left side P(targ=n|s_(0:t), obj_(1:N,0:t)) of the expression (1)represents a probability of the driver directing his or her gaze towardthe n-th gaze candidate under the assumption of the polar coordinates ofthe gaze LS from the time point 0 to the time point t and the centerangle and angle width of each of the first to N-th gaze candidates fromthe time point 0 to the time point t.

The right side p(s_(t)|targ=n, obj_(n,t)) of the expression (1)represents a probability density of the gaze s_(t) on the assumptionthat the driver is directing his or her gaze toward an object n. Whenthe gaze candidate is one from among the rearview mirror RVM, the frontwindshield FWS, the left side view mirror LSM, the left windshield LWS,the right side view mirror RSM, the right windshield RWS, and the dashboard DB, the probability density p(s_(t)|targ=n, obj_(n,t)) of thedriver directing his or her gaze toward the gaze candidate is defined bya uniform distribution. On the other hand, when the gaze candidate isthe object OB, the probability density p(s_(t)|targ=n, obj_(n,t)) isdefined by the following expression (2) representing a multivariatenormal distribution.

[Math.2] $\begin{matrix}{{p( {{{s_{t}❘{targ}} = n},{obj}_{n,t}} )} = {\frac{1}{2{{\pi\sigma}_{\varphi}( \Phi_{{obj}_{n,t}} )}{\sigma_{\theta}( \Theta_{{obj}_{n,t}} )}}{\exp( {{- \frac{( {\varphi_{s_{t}} - \varphi_{{obj}_{n,t}}} )^{2}}{2{\sigma_{\varphi}( \Phi_{{obj}_{n,t}} )}^{2}}} - \frac{( {\theta_{s_{t}} - \theta_{{obj}_{n,t}}} )^{2}}{2{\sigma_{\theta}( \Theta_{{obj}_{n,t}} )}^{2}}} )}}} & (2)\end{matrix}$

The expression (2) represents a normal distribution in which the centerangle (φ_(obj_n,t), θ_(obj_n,t)) of the object OB is an average and theangle width (Φ_(obj_n,t), Θ_(obj_n,t)) is a standard deviation. In otherwords, the expression (2) evaluates a higher probability value of thedriver directing his or her gaze toward the object OB as the directions_(t)(φ_(s_t), θ_(s_t)) of the gaze LS becomes closer to the centerdirection (φ_(obj_n,t), θ_(obj_n,t)) of the object OB. In the expression(2), the variances σ_(φ)(Φ_(obj_n,t)) and σ_(θ)(Φ_(obj_n,t)) are definedby the following expressions (3) and (4) using parameters Φ_(base) andΘ_(base) defined as fixed values in advance.

[Math. 3]

σ_(φ)(φ_(obj) _(n,t) )=Φ_(obj) _(n,t) /ψ_(base)  (3)

[Math. 4]

σ_(θ)(Θ_(obj) _(n,t) )=Θ_(obj) _(n,t) /Θ_(base)  (4)

Further, in the right side of the expression (1), P(targ=n|s_(0:t-1),obj_(1:N,0:t-1) represents a previous probability of the driverdirecting his or her gaze toward the n-th gaze candidate, and an initialvalue thereof is a fixed value or designed in advance according to thenumber of objects existing near the vehicle M. In this manner, it ispossible to calculate the current probability value in a robust mannerin consideration of the previous probability value by multiplying theprevious probability value P(targ=n|s_(0:t-1), obj_(1:N0:t-1)) by theprobability density p(s_(t)|targ=n, obj_(n,t)). Further, in the rightside of the expression (1), p(s_(t)|s_(0:t-1), obj_(1:N,0:t)) representsa normalization parameter for adjusting the probability value, and isdefined so as to satisfy the following expression (5).

[Math.5] $\begin{matrix}{{\sum\limits_{n = 1}^{N}{P( {{{targ} = {n❘s_{0:t}}},{obj}_{{1:N},{0:t}}} )}} = 1} & (5)\end{matrix}$

When the gaze determination unit 130 has estimated that the driver isdirecting his or her gaze toward one of the plurality of gaze candidatesfor the first predetermined period or more, the gaze determination unit130 determines that the driver visually recognizes the gaze candidate.FIG. 5 is a diagram for describing a method of determining that thedriver visually recognizes a gaze candidate by the gaze determinationunit 130. As illustrated in FIG. 5 , the gaze determination unit 130first recognizes that the probability value corresponding to a gazecandidate is equal to or larger than a threshold value p_(th) at a timepoint t1. After that, when the probability value is equal to or largerthan the threshold value p th until a time point t2 after elapse of afirst predetermined period T1 since the time point t1, the gazedetermination unit 130 determines that the driver visually recognizesthe gaze candidate. After that, the probability value is equal to orlarger than the threshold value p th until the time point t3, and thusthe gaze determination unit 130 determines that the driver visuallyrecognizes the gaze candidate continuously.

After that, the time point t3 has passed, and the probability valuebecomes smaller than the threshold value p_(th). However, even when theprobability value has become smaller than the threshold value p_(th),the gaze determination unit 130 keeps a determination result indicatingthat the driver visually recognizes the gaze candidate for a secondpredetermined period T2. This is because when the driver has directedhis or her gaze toward a gaze candidate for at least the firstpredetermined period T1, even after the driver has turned his or hergaze away from the gaze candidate, the driver is estimated to recognizethe state of the gaze candidate for a certain period. In this manner, bykeeping the determination result for a fixed period even after thedriver has turned his or her gaze away from a gaze candidate, thedriving assistance unit 140 described later can execute drivingassistance according to actual recognition by the driver. Morespecifically, by keeping the determination result for a fixed period,the driving assistance unit 140 can provide driving assistance that hasreduced a deviation between actual recognition by the driver and thelevel of driving assistance and unnecessary warning by drivingassistance due to the deviation.

The gaze determination unit 130 may change the lengths of the firstpredetermined period and the second predetermined period to be used fordetermining visual recognition according to the type of the gazecandidate. For example, the gaze determination unit 130 may decrease thefirst predetermined period or increase the second predetermined periodwhen the gaze candidate is used for checking the front side or lateralside of the vehicle M as in the case of the front windshield FWS, theleft windshield LWS, the right windshield RWS, and the dash board DBcompared with when the gaze candidate is used for checking the back sidefo the vehicle M as in the case of the rearview mirror RVM, the leftside view mirror LSM, and the right side view mirror RSM.

Further, for example, the gaze determination unit 130 may increase thesecond predetermined period as the period in which the driving isdirecting his or her gaze toward a gaze candidate, that is, the period(period between time point t1 and time point t3 in FIG. 5 ) in which theprobability value is equal to or larger than the threshold value p thbecomes longer. Further, for example, the gaze determination unit 130may integrate the probability value (or difference between probabilityvalue and threshold value p th) over a period in which the probabilityvalue is equal to or larger than the threshold value p th, and increasethe second predetermined period as the integrated value becomes larger.

Further, for example, the gaze determination unit 130 may increase thesecond predetermined period as the number (number of times) of objectsvisually recognized by the driver becomes larger among a plurality ofobjects recognized by the object recognition unit 110 in the past withina fixed period. In this case, the gaze determination unit 130 mayfurther increase the second predetermined period as the risk of anobject visually recognized by the driver becomes higher among theplurality of objects recognized by the object recognition unit 110 inthe past within a fixed period. For example, the gaze determination unit130 may calculate a TTC (time to collision) between the vehicle M and anobject, and increase the second predetermined period by determining therisk of a visually recognized object to be higher as the TTC between thevehicle M and the object visually recognized by the driver becomessmaller.

Further, for example, the gaze determination unit 130 may calculate alap ratio between the vehicle M and an object, and increase the secondpredetermined period by determining the risk of a visually recognizedobject to be higher as the lap ratio between the vehicle M and theobject visually recognized by the driver becomes larger. FIG. 6 is adiagram for describing a method of calculating the second predeterminedperiod in consideration of the lap ratio by the gaze determination unit130. For example, the gaze determination unit 130 calculates, as a lapamount β, the overlapping amount (distance in vehicle width direction inthe example of FIG. 6 ) between a region obtained by extending the widthα of the vehicle M in the travel direction and a region of a pedestrianserving as the object OB. The gaze determination unit 130 multiplies thedivision of the lap amount β by the vehicle width α by 100 ((β/α)×100)to obtain the lap ratio [%]. The gaze determination unit 130 mayincrease the second predetermined period by determining the risk of avisually recognized object to be higher as the lap ratio between thevehicle M and the object visually recognized by the driver becomeslarger. With this processing, it is possible to determine a period inwhich to keep the determination result according to actual recognitionby the driver.

The driving assistance unit 140 executes driving assistance for thevehicle M on the basis of the result of determination by the gazedetermination unit 130. More specifically, the driving assistance unit140 executes driving assistance so as to reduce the degree of alertnessfor an object when the gaze determination unit 130 has determined thatthe driver visually recognizes a gaze candidate, and visual recognitionof the gaze candidate enables recognition of the object included in thegaze candidate.

FIG. 7 is a diagram for describing driving assistance to be executed bythe driving assistance unit 140. FIG. 7 shows, as an example, a scene inwhich the driving assistance unit 140 executes driving assistance in asituation in which the driver visually recognizes the left side viewmirror LSM. In FIG. 7 , the reference symbols M1 and M2 represent othervehicles, and the reference symbol LSM_R represents a range of objectsthat can be recognized by visually recognizing the left side view mirrorLSM. The recognition range LSM_R includes another vehicle M1 but doesnot include another vehicle M2. In other words, it is expected that thedriver recognizes another vehicle M1 by visually recognizing the leftside view mirror LSM, whereas the driver does not recognize anothervehicle M2.

Thus, the driving assistance unit 140 executes driving assistance so asto reduce the degree of alertness for another vehicle M1 (increase thedegree of alertness for another vehicle M2). FIG. 8 is a diagram fordescribing driving assistance to be executed by the driving assistanceunit 140. As illustrated in FIG. 8 , the driving assistance unit 140displays, on the HMI 30, a message for suggesting to drive the vehicle Mwhile watching out for another vehicle M2 (motorcycle) not included inthe recognition range LSM_R. Further, for example, the drivingassistance unit 140 may cause the speaker to output a message forsuggesting to drive the vehicle M while watching out for another vehicleM2. Further, for example, the driving assistance unit 140 may output acommand value to the steering device 220 so as to drive the vehicle Maway from another vehicle M2 not included in the recognition rangeLSM_R. In this manner, it is possible to execute driving assistanceaccording to actual recognition of the driver by decreasing the degreeof alertness of driving assistance only for an object included in arange recognized by visually recognizing a gaze candidate.

[Flow of Operations]

Next, description is given of a flow of operations to be executed by thedriving assistance device 100 with reference to FIG. 9 . FIG. 9 is aflow chart illustrating an exemplary flow of operations to be executedby the driving assistance device 100. The processing of the flow chartillustrated in FIG. 9 is repeatedly executed by the driving assistancedevice 100 in specific control cycles during traveling of the vehicle M.

First, the object recognition unit 110 recognizes an object existingnear the vehicle M on the basis of the result of recognition by theobject recognition unit 16 (Step S100). Next, the gaze estimation unit120 calculates, for each of a plurality gaze candidates including therecognized object, a probability value of the driver directing his orher gaze toward the gaze candidate, and estimates a gaze candidate forwhich the calculated probability value is equal to or larger than thethreshold value p th as the gaze candidate toward which the driver isdirecting his or her gaze (Step S102).

Next, the gaze determination unit 130 determines, for the gaze candidatetoward which the driver is estimated to be directing his or her gaze bythe gaze estimation unit 120, whether or not the period in which thecalculated probability value is equal to or larger than the thresholdvalue p_(th) is equal to or longer than the first predetermined periodT1 (Step S104). When it is determined that the period in which thecalculated probability value is equal to or larger than the thresholdvalue p th is shorter than the first predetermined period T1, the gazedetermination unit 130 returns the processing to Step S102, and the gazedetermination unit 130 estimates a gaze candidate toward which thedriver is directing his or her gaze again.

On the other hand, when it is determined that the period in which thecalculated probability value is equal to or larger than the thresholdvalue p th is equal to or longer than the first predetermined period T1,the gaze determination unit 130 determines that the driver visuallyrecognizes the gaze candidate (Step S106). Next, the gaze determinationunit 130 determines whether or not the driver has turned his or her gazeaway from the gaze candidate (in other words, whether or not theprobability value becomes smaller than the threshold value p_(th)) (StepS108). When it is determined that the driver has not turned his or hergaze away from the gaze candidate, the gaze determination unit 130returns the processing to Step S106, and the gaze determination unit 130determines that the driver visually recognizes the gaze candidatecontinuously.

When it is determined that the driver has turned his or her gaze awayfrom the gaze candidate, the gaze determination unit 130 keeps thedetermination result indicating that the driver visually recognizes thegaze candidate for the second predetermined period T2 since the driverhas turned his or her gaze away from the gaze candidate (Step S110).Next, the driving assistance unit 140 executes driving assistance forthe vehicle M so as to reduce the degree of alertness for the gazecandidate (more specifically, the degree of alertness for an objectincluded in the range recognized by visually recognizing the gazecandidate) during the second predetermined period T2 (Step S112). Inthis manner, the processing of this flow chart is finished.

According to the embodiment described above, the driving assistancedevice recognizes an external object near a mobile object, estimatestoward which gaze candidate an occupant of the mobile object isdirecting his or her gaze from among a plurality of gaze candidatesincluding the recognized external object, determines that the occupantvisually recognizes a gaze candidate when the occupant is estimated todirect his or her gaze toward the gaze candidate for a firstpredetermined period or more, keeps the determination result indicatingthat the occupant visually recognizes the gaze candidate for a secondpredetermined period since the occupant has turned his or her gaze awayfrom the gaze candidate, and executes driving assistance for the mobileobject so as to reduce the degree of alertness for the gaze candidatedetermined to be visually recognized by the occupant. With this, it ispossible to flexibly execute driving assistance according to thedirection or period of gaze of the occupant of a mobile object.

The embodiment described above can be represented in the followingmanner.

A driving assistance device including a storage medium storingcomputer-readable commands and a processor connected to the storagemedium, the processor executing the computer-readable commands to:recognize an external object near a mobile object; estimate toward whichgaze candidate an occupant of the mobile object is directing his or hergaze from among a plurality of gaze candidates including gaze candidatesexisting on a structure of the mobile object and the external object;determine that the occupant visually recognizes a gaze candidate fromamong the plurality of gaze candidates when the occupant is estimated todirect his or her gaze toward the gaze candidate for a firstpredetermined period or more; execute driving assistance for the mobileobject based on a determination result; and keep, when it is estimatedthat the occupant has turned his or her gaze away from the gazecandidate after determining that the occupant visually recognizes thegaze candidate, the determination result, which indicates that theoccupant visually recognizes the gaze candidate determined to be gazedat, for a second predetermined period since a time point at which theoccupant is estimated to have turned his or her gaze away from the gazecandidate.

This concludes the description of the embodiment for carrying out thepresent invention. The present invention is not limited to theembodiment in any manner, and various kinds of modifications andreplacements can be made within a range that does not depart from thegist of the present invention.

What is claimed is:
 1. A driving assistance device comprising a storagemedium storing computer-readable commands and a processor connected tothe storage medium, the processor executing the computer-readablecommands to: recognize an external object near a mobile object; estimatetoward which gaze candidate an occupant of the mobile object isdirecting his or her gaze from among a plurality of gaze candidatesincluding gaze candidates existing on a structure of the mobile objectand the external object; determine that the occupant visually recognizesa gaze candidate from among the plurality of gaze candidates when theoccupant is estimated to direct his or her gaze toward the gazecandidate for a first predetermined period or more; and execute drivingassistance for the mobile object based on a determination result,wherein when the processor has estimated that the occupant has turnedhis or her gaze away from the gaze candidate after determining that theoccupant visually recognizes the gaze candidate, the processor keeps thedetermination result, which indicates that the occupant visuallyrecognizes the gaze candidate determined to be gazed at, for a secondpredetermined period since a time point at which the occupant isestimated to have turned his or her gaze away from the gaze candidate.2. The driving assistance device according to claim 1, wherein theprocessor estimates toward which gaze candidate the occupant of themobile object is directing his or her gaze from among the plurality ofgaze candidates based on a degree of match between the gaze and a gazeprobability distribution representing probabilities of gaze toward eachgaze candidate.
 3. The driving assistance device according to claim 2,wherein the processor calculates the degree of match based on the gazerepresented by polar coordinates centered around a head of the occupantof the mobile object, and a center angle and angle width of the gazecandidate represented by polar coordinates centered around the head, andestimates toward which gaze candidate the occupant of the mobile objectis directing his or her gaze from among the plurality of gazecandidates.
 4. The driving assistance device according to claim 1,wherein the processor changes a length of the second predeterminedperiod according to types of the plurality of gaze candidates.
 5. Thedriving assistance device according to claim 1, wherein the processorincreases the second predetermined period as a period in which theoccupant is determined to visually recognize the gaze candidate becomeslonger.
 6. The driving assistance device according to claim 1, whereinthe processor increases the second predetermined period as the number oftimes the occupant visually recognizes the recognized external objectbecomes larger.
 7. The driving assistance device according to claim 2,wherein the processor changes a length of the second predeterminedperiod according to a level of the calculated degree of match.
 8. Thedriving assistance device according to claim 1, wherein when it isdetermined that the occupant visually recognize the gaze candidate andvisual recognition of the gaze candidate enables recognition of anobject included in the gaze candidate, the processor execute the drivingassistance so as to reduce a degree of alertness for the object.
 9. Adriving assistance method to be executed by a computer, the drivingassistance method comprising: recognizing an external object near amobile object; estimating toward which gaze candidate an occupant of themobile object is directing his or her gaze from among a plurality ofgaze candidates including gaze candidates existing on a structure of themobile object and the external object; determining that the occupantvisually recognizes a gaze candidate from among the plurality of gazecandidates when the occupant is estimated to direct his or her gazetoward the gaze candidate for a first predetermined period or more;executing driving assistance for the mobile object based on adetermination result; and keeping, when it is estimated that theoccupant has turned his or her gaze away from the gaze candidate afterdetermining that the occupant visually recognizes the gaze candidate,the determination result, which indicates that the occupant visuallyrecognizes the gaze candidate determined to be gazed at, for a secondpredetermined period since a time point at which the occupant isestimated to have turned his or her gaze away from the gaze candidate.10. A non-transitory computer-readable storage medium storing a programfor causing a computer to: recognize an external object near a mobileobject; estimate toward which gaze candidate an occupant of the mobileobject is directing his or her gaze from among a plurality of gazecandidates including gaze candidates existing on a structure of themobile object and the external object; determine that the occupantvisually recognizes a gaze candidate from among the plurality of gazecandidates when the occupant is estimated to direct his or her gazetoward the gaze candidate for a first predetermined period or more;execute driving assistance for the mobile object based on adetermination result; and keep, when it is estimated that the occupanthas turned his or her gaze away from the gaze candidate afterdetermining that the occupant visually recognizes the gaze candidate,the determination result, which indicates that the occupant visuallyrecognizes the gaze candidate determined to be gazed at, for a secondpredetermined period since a time point at which the occupant isestimated to have turned his or her gaze away from the gaze candidate.